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FOREWORD 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences'  Infantry  Forces  Research  Unit  has  over  the  past 
several  years  conducted  research  designed  to  improve  training  and 
performance  for  battalion  and  brigade  staff  officers.  This 
report  documents  the  design  and  development  of  13  courses  of 
instruction  delivered  for  the  Brigade  Battle  Staff  Training 
System  (BDE-BSTS) .  BDE-BSTS  is  a  set  of  13  training  packages  for 
brigade-level  staff  officers,  a  mixture  of  text  and  computer- 
based  instruction.  Developed  for  use  by  the  U.S.  Army  National 
Guard,  the  work  was  sponsored  by  the  Advanced  Research  Projects 
Agency  (ARPA)  as  an  advanced  technology  demonstration  under  the 
Simulation  in  Training  for  Advanced  Readiness  program.  The  work 
was  performed  by  BDM  Federal,  Inc. 

The  BSTS  concept  has  potential  for  enhancement  of  other 
training.  Adaptable  for  any  course  of  instruction,  the  BSTS 
model  can  become  a  cornerstone  for  individual  and  small-group 
training.  It  can  be  used  for  distance  learning  and  to  augment 
institutional  or  unit  training  for  newly  assigned  personnel.  The 
BSTS  program  has  been  briefed  to  personnel  throughout  Training 
and  Doctrine  Command  and  is  being  adapted  for  implementation  in 
the  Force  XXI  Training  Program. 


ZITA  M.  SIMUTIS  EDGAR  M.  JOHNSON 

Deputy  Director  Director 

(Science  and  Technology) 
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BRIGADE  BATTLE  STAFF  TRAINING  SYSTEM 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  Battle  Staff  Training  System  (BSTS)  was  designed  to 
address  an  identified  need  for  staff  functional  area  training  for 
maneuver  battalion  and  brigade  staff  officers.  Recognizing  that 
the  need  for  such  training  was  especially  critical  for  the 
Reserve  Component,  the  Advanced  Research  Projects  Agency  and  the 
U.S.  Army  National  Guard  supported  the  development  of  the 
prototype  BSTS  for  experimental  application  in  the  National  Guard 
under  the  program  Simulation  in  Training  for  Advanced  Readiness. 

Procedure : 

In  order  to  train  individual  staff  skills  through  a 
multimedia  approach  using  computer-based  instruction  complemented 
by  text-based  materials,  13  doctrinally  based  staff  training 
modules  were  developed.  Included  in  the  brigade  set  are 
interactive  software  and  text-based  supporting  materials  for  the 
Executive  Officer,  SI,  S2,  S3,  S3-Air,  S4,  S5,  Fire  Support 
Officer,  Signal  Officer,  Chemical  Officer,  Engineer,  and  Air 
Defense  Artillery  Officer.  An  additional  Common  Core  Course 
provides  information  used  by  all  staff  officers.  The  brigade 
courses  parallel  previously  developed  battalion  staff  courses. 

Findings : 

The  prototype  program,  on  a  486  PC  with  CD  ROM  with  specific 
software,  has  an  introductory  tutorial  to  teach  all  operational 
functions.  Each  course  module  provides  core  material  necessary 
for  the  officer  to  function  in  the  assigned  staff  position. 

There  are  practical  exercises,  diagnostic  pretests,  within-topic 
quizzes,  and  a  final  examination  to  measure  progress  and  provide 
feedback.  The  BSTS  permits  the  staff  officer  to  train  at  his  own 
pace  at  home,  in  the  unit,  or  at  the  armory. 

Utilization  of  Findings: 

While  the  Reserve  Component  need  provided  the  initial 
opportunity  for  development  of  a  prototype  entry-level  training 
system,  other  applications  are  apparent.  The  Force  XXI  Training 
Program  at  Fort  Knox  has  sponsored  an  expanded  BSTS  with 
commander  modules,  comprehensive  examinations,  and  remedial 
lessons.  Additional  research  at  the  Joint  Readiness  Training 
Center  Leader  Training  Program  may  further  enhance  the  program 
for  training  staff  synchronization  and  integration. 
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BRIGADE  BATTLE  STAFF  TRAINING  SYSTEM 


Introduction 

This  report  documents  the  development  of  the  courses  of 
instruction  for  the  Brigade  Battle  Staff  Training  System  (BDE- 
BSTS) .  BDE-BSTS,  a  set  of  functional  area  training  packages  for 
Brigade  level  staff  officers,  is  a  combination  of  text  and 
computer-based  instruction  (CBI) .  The  packages  include  CD-ROM 
based  programs,  supplemental  text-based  instruction,  and  an 
embedded  instructional  and  training  management  system.  Designed 
primarily  for  local  area  networks,  or  wide  area  network  linked 
training _ systems,  the  BDE-BSTS  was  developed  for  use  by  the  U.S. 
Army  National  Guard  (ARNG) .  The  work  was  performed  by  BDM, 
Federal,  Inc,,  at  Fort  Benning,  Georgia  from  August  1994  through 
February  1996  under  contract  to  the  Advanced  Research  Projects 
Agency  (ARPA)  and  the  Army  Research  Institute  (ARI) ,  Infantry 
Forces  Research  Unit. 


Background 

The  ARPA  Land  Systems  Office  and  the  ARNG  Advanced 
Distributed  Simulation  Program  have  focused  on  the  unique 
training  needs  of  the  Reserve  Components.  Part  of  the  training 
challenge  for  ARNG  combat  arms  staff  members  is  due  to  conflicts 
with  other  required  duties .  To  help  alleviate  this  problem,  the 
ARPA,  ARNG,  and  ARI  initiated  development  of  distributed,  multi- 
media  (paper -based  and  computer-based)  instruction  for  staff 
personnel.  A  prototype  thirteen  course  battalion  battle  staff 
training  system  (BN-BSTS)  was  developed  (Andre'  &  Salter,  1995) . 
This  BSTS  prototype  was  then  expanded  to  include  the  increased 
horizontal  and  vertical  coordination  responsibilities  of  the 
brigade  staff.  This  report  documents  that  expansion,  the 
prototype  BDE-BSTS.  The  resulting  two  BSTS  training  systems 
provide  prototype  training  programs  to  cover  individual  battalion 
and  brigade  staff  functional  areas  and  those  tasks  required  to 
prepare  staffs  for  collective  battle  staff  tasks. 

The  BSTS  program,  sponsored  under  the  ARPA  program  umbrella 
of  Simulation  in  Training  for  Advanced  Readiness  (SIMITAR) ,  is 
coordinated  with  three  other  programs:  Simulation-Based  Mounted 
Brigade  Training  Program  (SIMBART) ,  Simulation-Based  Multiechelon 
Training  for  Armor  Units  (SIMUTA)  and  Combat  Service  Support 
(CSS)  Training  System  Development  for  the  National  Guard.  Both 
BSTS  programs  applied  innovative  research,  application,  and 
technologies  of  instructional  design,  computer  science,  and 
multi-media  training  strategies.  The  BDE-BSTS,  like  the  BN-BSTS, 
will  be  distributed  to  two  ARNG  test  brigades  which  will 
participate  in  separate  evaluations  at  the  National  Training 
Center  (NTC) . 
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Prior  Research 


The  need  for  formal  battle  staff  training,  especially  for 
units  training  for  rotations  to  a  Combat  Training  Center  (CTC) , 
became  evident  from  ARI's  research  on  home  station  determinants 
of  effective  unit  performance  at  the  CTCs  (Holz,  Hiller  &  McFann, 
1994) .  The  analysis  of  unit  performance  at  the  NTC  and  the  Joint 
Readiness  Training  Center  (JRTC)  indicated  that  special  training 
for  the  battle  staff  (especially  orders  drills)  could  lead  to 
better  unit  performance. 

Research  that  focused  on  light  infantry  units  showed  that 
training  in  the  appropriate  functional  area  was  generally  not 
provided  prior  to  assignment  to  maneuver  battalion  staff 
(Thompson,  Thompson,  Pleban  &  Valentine,  1991) .  The  only  officer 
training  available  was  conducted  as  on-the-job  training.  As  a 
result,  the  battle  staff  did  not  recognize  deficiencies  in 
synchronization  of  critical  command  and  staff  activities  until 
after  a  rotation  was  under  way  or  completed. 

The  battle  staff  competency  issue  has  been  addressed  and 
research  conducted  on  the  problem  of  training  battle  staffs.  An 
interim  solution.  The  Commander's  Battle  Staff  Handbook  (Pleban, 
Thompson,  &  Valentine,  1993)  was  developed,  and  distributed  to 
both  Active  Component  (AC)  and  Reserve  Component  (RC)  personnel. 
The  handbook  describes  individual  battalion  battle  staff  duties, 
responsibilities,  and  references  for  key  staff /slice  officers  and 
the  command  sergeant  major.  It  was  developed  from  staff  material 
provided  by  TRADOC  branch  schools,  interviews  with  subject  matter 
experts,  and  from  the  assessment  of  unit  performance  by  observers 
at  the  CTCs .  This  handbook  was  the  primary  base  document  for  the 
BN-BSTS  courses.  A  similar  product.  Brigade  Commanders  Battle 
Staff  Handbook  (Valentine  &  Andre',  in  preparation),  served  as 
the  foundation  for  the  BDE-BSTS. 

BDE-BSTS,  like  BN-BSTS,  also  incorporated  the  concept  of 
organizational  competence  (Olmstead,  1992).  Olmstead's  tenets 
became  training  objectives  in  a  common  core  course  of  instruction 
and  were  integrated  into  the  remaining  courses .  The  intent  was 
to  ensure  that  officers  and  the  staff  possess  the  skills  required 
to  fully  synchronize  individual  efforts  into  the  collective  staff 
effort.  This  should  lead  to  improved  performance  during  the 
planning,  preparation  and,  to  a  lesser  extent,  execution  phases 
of  a  given  mission  or  individual/collective  tasks. 
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Method 


The  primary  developers  of  the  instructional  materials  were  a 
team  of  subject  matter  experts  (SMEs)  with  extensive  experience 
serving  on  AC  and  RC  Brigade  battle  staffs,  in  combat  and  at  the 
CTCs.  They  determined  the  individual  training  objectives  for 
each  course,  using  as  primary  reference  the  Brigade  Commander ' s 
Battle  Staff  Handbook  (Valentine  &  Andre',  in  preparation)  and 
doctrinal  materials .  The  handbook  and  the  preceding  battalion 
project  provided  the  foundation  for  the  front  end  analysis  and 
task  lists.  SMEs  surveyed  existing  formal  courses.  Army  doctrinal 
publications  (Field  Manuals,  Mission  Training  Plans,  Technical 
Manuals,  Programs  of  Instruction  (POIs) ,  etc.),  and  related 
materials.  Additionally,  they  used  information  from  Center  for 
Army  Lessons  Learned  (CALL)  extracts,  trends,  reports,  and  data, 
and  CTC  After  Action  Reviews  (AARs) . 

POIs  were  written,  following  the  accepted  instructional 
design  models  of  the  systems  approach  to  training  (Department  of 
the  Army,  1982  &  1995) .  Additional  models  were  created  for  the 
design,  development  and  delivery  of  text-based  and  CBI  training. 
State-of-the-art  hardware  and  software  from  the  entertainment  and 
computer  industries  and  CBI  software  development  firms  were 
utilized. 


Instructional  Overview 

CBI  is  the  basis  for  this  training  system.  Staff  officer 
materials  are  on  CD  ROM  in  conjunction  with  accompanying  text- 
based  materials.  (Technical  specifications  are  provided  at 
Appendix  B.)  Decisions  on  presentation  media  considered  cost  and 
training  effectiveness  of  existing  and  new  materials.  Also,  CBI 
presentation  supplemented  with  text-based  and  companion  doctrinal 
publications  provided  an  inherently  more  interesting 
instructional  program  than  existing  text-based  materials  alone. 
Maintaining  student  interest  was  deemed  essential  because  of  the 
independent  nature  of  the  training  program. 

The  CBI  format  was  designed  to  provide  the  diagnostic, 
assessment,  and  performance  measurement  components  of  the  system 
as  well  as  provide  the  primary  instructional  medium.  Although 
text-based  materials  supplement  CBI,  exercises  and  tutorials  and 
all  examinations  are  presented  through  the  computer. 

Additionally,  electronic  bulletin  boards  or  a  "chat" 
capability  (provided  by  two  Shiva  net  modems)  permit  access  to 
other  staff  members.  The  staff  can  conduct  outside  class 
meetings  for  problem  solving  and  discussion  of  course  content . 
This  can  assist  in  the  preparation  for  collective  tasks. 
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Courses  of  Instruction 


The  thirteen  courses  of  instruction  for  BDE-BSTS  are:  Common 
Core,  Executive  Officer  (XO) ,  Sl/Chaplain,  S2,  S3,  S3  Air,  S4, 

S5,  Fire  Support  Officer  (FSO) ,  Engineer,  Air  Defense  Artillery 
Officer  (ADA) ,  Signal  Officer  (SIGO)  and  Chemical  Officer 
(CHEMO) .  Each  of  the  courses  is  composed  of  a  number  of 
subjects.  Subjects  are  groups  of  lessons  which  are  composed  of 
topics  that  cover  a  major  learning  objective.  Figure  1,  Model 
Course  Map,  shows  an  example  of  this  structure.  Within  the 
Brigade  Common  Core  Course,  there  are  four  subjects.  Number 
Three,  Staff  Functions,  contains  three  lessons  (Use  of  Graphics, 
Military  Briefings,  and  Staff  Integration) .  Each  topic  contains 
several  lessons.  In  the  example  model,  a  check  mark  within  a  box 
indicates  that  the  topics  are  presented  through  computer  based 
instruction. 
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Figure  1 .  Model  course  map 
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Lessons  take  one  or  two  hours  to  complete,  depending  on  the 
student.  The  training  management  system.  Environment  for  Multi- 
Media  Interactive  Instruction  (EMMii) ,  allows  the  commander, 
executive  officer,  training  officer,  or  other  designated 
individual  to  schedule  courses,  monitor  the  progress  of 
individual  officers,  and  manage  the  authoring  of  revisions  to 
courses  of  instruction.  (Appendix  C  provides  EMMii  background 
information. ) 


Instructional  Concept 

The  combination  of  CBI  and  text  materials  included  graphics, 
still  photographs,  audio,  text,  and  full  motion  video  with  audio. 
Decisions  on  specific  application  of  media  were  based  on 
selection  of  what  was  deemed  the  most  effective  medium  for 
teaching  the  learning  objective.  CBI  was  used  where  it  appeared 
clearly  superior,  especially  to  present  high  order  cognitive 
tasks  and  complex  tasks  that  require  coordination  or 
synchronization  with  other  tasks.  Practical  applications 
(exercises)  of  skills  or  knowledge  presented  were  also  contained 
in  the  CBI.  Instruction  focuses  on  mastery  of  basic  skills  and 
knowledge  followed  by  application  of  critical  principles  to 
ensure  the  student  can  apply  the  task  in  a  realistic  environment. 
All  diagnostics,  assessment,  measurement  and  feedback  were 
included  in  CBI  to  provide  trainers  with  the  capability  of 
tracking  student  progress. 

The  instructional  concept  places  the  student  in  control  of 
the  pace  and  sequence  of  instruction.  Feedback  provided  to  the 
student  informs  him  or  her  if  the  task  has  been  mastered. 

Computer  tracking  of  status  permits  trainers  in  the  chain  of 
command  to  recommend  remedial  training  where  needed. 

Development  of  Instructional  Content 

Courses  of  instruction  were  designed  for  the  brigade  battle 
staff,  which  consists  of  the  commander's  primary  and  special 
staff  as  well  as  the  Combat  Support  (CS)  and  Combat  Service 
Support  (CSS)  slices  that  habitually  work  with  the  staff,  using 
the  same  process  as  was  used  in  the  development  of  the  battalion 
program.  As  a  result  of  the  program  front-end  analysis  (FEA) , 
one  course  of  instruction  was  added  to  the  brigade  list,  the  S5, 
or  Civil  Affairs  Officer.  The  courses  for  the  SI  and  the 
Chaplain  were  combined  into  one  (SI /Chaplain)  for  the  brigade 
program.  The  needs  assessment,  conducted  to  determine  the  tasks 
to  be  trained  for  the  selected  battle  staff  officers,  was 
conducted  for  each  course,  and  published  as  an  overall  brigade 
task  list  (Andre',  in  preparation). 
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One  person  was  assigned  primary  responsibility  for  the 
design  and  development  of  each  course  of  instruction.  The 
contractor  SME,  in  conjunction  with  an  instructional  designer, 
applied  a  functional  job  analysis  to  develop  the  training 
objectives  and  corresponding  performance  measures  to  be  trained. 
Figure  2  shows  this  process.  Within  the  figure,  several 
abbreviations  are  used  for  economy  of  space.  The  abbreviated 
words  are:  Battlestaff  Handbook  (BSHB) ;  Tasks,  Conditions  and 
Standards  (TCS) ;  Computer  based  instruction  (CBI) ;  Practical 
Exercises  (PEs) ;  Audio  Visual  (AV) ;  Quality  Assurance  (QA) ; 
Formative  Evaluation  (FE) ,  Phases  (PH)  I,  II,  III;  Contractor 
Military  Expert  (CME) ;  Subject  Matter  Expert  (SME) ;  Target 
Population  (TGT  POP) ;  and  Training  Support  Package  (TSP) . 

For  each  course,  the  authors  surveyed  and  documented 
relevant  doctrinal  and  training  literature,  lessons  learned,  and 
applicable  Army  formal  courses  and  instructional  materials. 
Following  the  literature  review,  they  applied  the  training 
development  model,  culminating  in  a  final  version  of  the  text- 
based  instruction  with  adjunctive  materials  and  tests.  Quality 
assurance  (QA)  of  materials  was  accomplished  in  house,  by  the 
contractor  team,  by  SMEs  for  content,  and  by  an  instructional 
designer  for  application  of  sound  instructional  principles. 

After  QA  on  the  overall  instruction,  the  most  complex 
cognitive  tasks,  as  well  as  those  determined  to  require 
integration  and  synchronization  with  other  complex  tasks,  were 
designated  for  application  of  CBI.  Consequently,  storyboards, 
with  attached  media,  were  provided  to  the  CBI  authors  for 
development  into  CBI  with  specified  training  application, 
sequence,  branching,  and  selected  media. 

Each  of  the  courses  also  has  a  listing  of  associated  related 
professional  reference  materials  (e.g.,  technical  manuals), 
needed  to  complete  the  instruction.  Thus,  when  the  course  is 
completed,  the  staff  officer  should  be  technically  prepared  to 
perform  the  required  staff  function,  and  should  also  know  to 
refer,  as  required,  to  doctrinal  publications  for  specific 
application  of  functional  area  and  staff  skills. 
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BATTLE  STAFF  TRAINING  SYSTEM 
TRAINING  DEVELOPMENT  MODEL 
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Figure  2 .  Training  development  model 
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Training  Management 


Management  of  the  training  used  a  modification  of  the  system 
Environment  for  Multi-Media  Interactive  Instruction  (EMMii) 
developed  for  use  with  the  BN-BSTS,  BDE-BSTS  and  CSS-BSTS 
programs  and  designed  specifically  to  enhance  the  instructor- 
independent  nature  of  distance  learning  (see  Appendix  C  for 
details  on  EMMii) .  The  EMMii  is  an  enabling  environment  for 
multi-media  computer-based  training  development  and  delivery.  It 
comprises  a  system  that  allows  trainers  or  administrators  to 
schedule  student  training,  monitor  progress,  receive  status 
reports,  print  results,  analyze  measurement  data  for  course 
enhancement,  and  modify  courseware  on-line  (while  students  are 
training  on  previous  versions  of  the  course) . 

The  Windows  for  Workgroups  (™)  environment  provides  a 
familiar  look  and  feel  across  the  training  media.  A  model  of  the 
hardware  architecture  is  shown  in  Figure  3 .  EMMii  uses 
Microsoft's  Access  (™)  database  to  manage  the  student's 
progression  through  the  course  of  instruction.  Course  templates 
standardize  the  recording  of  student  progress  within  lessons  for 
posting  to  the  data-base.  This  allows  the  trainer  to  track  the 


Within  the  Armory  or  Learning  Center 


WAN  OR,  Stand  Alone 


Figure  3.  EMMii  hardware  architecture. 
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overall  progress  of  the  entire  staff  as  they  progress  in  their 
training. 


Authoring  of  Courseware 


CBI  development  used  commercial,  off-the-shelf,  software. 
Icon  Author  6.0  (’”)  was  used  to  develop  the  CBI,  templates,  and 
standards.  It  has  previously  been  accepted  as  meeting  the 
selection  criteriafor  an  authoring  system  (Park  &  Seidel,  1989) 
and  ensures  compatibility  with  the  previously  developed  BN-BSTS 
and  CSS  courses  of  instruction.  The  Icon  Author  package  proved 
relatively  easy  to  learn,  formal  training  was  readily  accessible, 
and  experienced  users  were  available.  Templates  were  developed 
to  include  learning  strategies,  techniques  for  question  and 
answer  sessions,  testing  criteria,  screen  design,  navigation 
through  the  courses  of  instruction,  help  screens,  job  aids,  and 
screen  layout.  (Appendix  D  contains  information  on  these 
Instructional  Standards.) 

CBI  development  required  three  skills  of  the  developers: 
subject  matter  expertise,  knowledge  of  instructional  design,  and 
computer  literacy  skills.  The  development  team  used  personnel 
who  possessed  both  instructional  design  and  CBI  design  skills  to 
prepare  storyboards  for  future  coding  into  CBI.  Authoring 
systems  were  available  at  several  locations  to  ensure  QA  of  all 
CBI.  Most  of  the  templates  and  Smart  Objects  developed  for  use 
were  developed  specifically  to  implement  the  design  standards  of 
the  BDE-BSTS  program.  Additionally,  templates  and  Smart  Objects 
were  developed  for  the  diagnostic,  performance  measurement  and 
assessment  component  of  the  system  to  enhance  the  overall 
learning.  Figure  4  provides  a  diagram  of  the  CBI  design  model. 
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*  Planned  enhancement 


Figure  4 .  CBI  design  model . 


Realism  in  Training 

The  multi-media  strategy  is  particularly  valuable  for 
providing  realism  and  retaining  interest  in  the  training.  This 
strategy  includes  the  application  of  tactical  scenarios  which  set 
the  context  for  the  lessons  and  practical  exercises .  Realism  was 
created  and  maintained  through  the  use  of  common  missions.  These 
scenarios,  with  accompanying  orders,  annexes  and  graphics,  were 
utilized  in  the  program  to  create  a  viable  context  for  training 
and  to  permit  the  use  of  tested  solutions.  This  allows  the 
student  to  review  and  evaluate  the  developed  solutions .  The 
common  scenarios  (BN-BSTS,  BDE-BSTS,  CSS-BSTS,  and  SIMBART)  were 
built  on  the  NTC-based  scenarios  developed  and  used  in  SIMUTA 
programs  (Brown,  1992).  Students  are  led  through  the  training 
context  (scenario)  applying  their  learned  skills  to  situations 
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presented.  Then,  as  appropriate,  they  are  provided  potential  or 
possible  solutions  against  which  to  compare  their  own  results. 

Quality  Assurance 

Quality  assurance  (QA)  was  integrated  throughout  the 
developmental  process,  as  shown  in  Figure  2  where  QA  is 
highlighted.  QA  for  BDE-BSTS  was  accomplished  by  SMEs  for  peer 
review  of  content,  the  instructional  designer  for  standards  and 
instructional  integrity,  the  program  manager  for  content  and 
standards,  and  computer  programmers  for  functionality  of  the 
code;  thereby  ensuring  quality  instruction  that  attained  the 
training  objective.  QA  occurred  throughout  the  process,  during 
development  of  training  objectives,  performance  measures,  text, 
tests  and  storyboards,  and  the  CBI  process. 


Results 

Thirteen  courses  were  designed  for  the  BDE-BSTS  program.  As 
shown  in  Table  1  with  the  estimated  niomber  of  hours  to  complete 
the  material,  the  courses  consisted  of:  Common  Core  (material 
common  to  all  positions).  Executive  Officer  (XO) ,  Sl/Chaplain 
(personnel) ,  S2  (intelligence) ,  S3  and  S3  Air  (operations) ,  S4 
(logistics),  S5  (Psychological  Operations  (PSYOP) /Civil  Affairs), 
Fire  Support  Officer  (FSO) ,  Engineer  (ENG),  Air  Defense  Artillery 
(ADCOORD) ,  Signal  (SIGO)  Officer  and  Chemical  (CHEMO)  Officer. 

The  estimated  numbers  of  CBI  course  hours  shown  in  Table  1  are 
good  approximations;  the  numbers  of  hours  spent  on  text  may  vary 
with  the  reading  speed  and  comprehension  of  the  student. 

The  BDE-BSTS  was  tested  through  a  formative  evaluation  (FE) 
process,  as  shown  in  Figure  5.  The  FE  included  Alpha  testing  by 
6  to  10  persons  who  are  SMEs  in  content,  computer  programming, 
and/or  instructional  design.  Alpha  testing  describes  the  process 
whereby  experts  in  each  of  several  disciplines  examines  the 
material  to  ensure  it  meets  specific  technical  standards  and 
criteria.  Testers  reviewed  the  text -based  materials,  CBI,  and  the 
complete  course  of  instruction  for  doctrinal  accuracy, 
instructional  soundness  and  functionality  of  the  computer 
programming . 
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Table  1 


Estimated  Number  of  Course  Hours 


COURSE 

TEXT 

CBI 

TOTAL 

Common  Core 

15.8 

12.2 

28.0 

XO 

25.7 

3.3 

29.0 

SI /Chaplain 

22.4 

4.8 

27.2 

S2 

15.0 

10.0 

25.0 

S3 

30.0 

8.0 

38.0 

S3  Air 

21.7 

8.3 

30.0 

S4 

19.1 

2.9 

22.0 

S5 

10.9 

3.1 

14.0 

FSO 

44.0 

6.0 

50.0 

Engineer 

21.4 

2.1 

23.5 

ADCOORD 

17.9 

10.1 

28.0 

SIGO 

3.6 

4.4 

8.0 

CHEMO 

24.7 

4.2 

28.9 

TOTALS 

272.2 

79.3 

351.6 
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*  Recommended,  not  implemented,  due  to  constraints. 


Figure  5 .  Testing  process  model 
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Testing  and  Evaluation 


Alpha  testing  of  the  text  was  initially  conducted  by  2-3 
contractor  SMEs  with  expertise  in  the  specific  battle  staff 
position  being  trained.  The  text  was  also  reviewed  by  2-3  SMEs 
from  Fort  Benning  and  other  TRADOC  proponents .  Alpha  testing  of 
CBI  was  conducted  by  computer  programmers  at  Fort  Benning,  GA  to 
ensure  the  proper  operation  of  all  codes . 

Beta  testing  refers  to  a  limited  user  test,  one  conducted  by 
a  sample  of  the  population  for  which  the  training  is  intended. 

The  Beta  test  of  each  course  of  instruction  was  completed  prior 
to  finalization  of  the  instructional  program.  Beta  test  results 
included  specific  and  overall  evaluations  of  each  of  the  courses 
of  instruction.  Of  the  12  statements  to  be  rated  by  every  test 
officer  on  a  post  course  questionnaire,  most  all  indicated 
approval  of  and  positive  response  to  the  courses.  Most  of  the 
negative  comments  were  related  to  the  students'  desires  that  more 
of  the  lessons  be  in  CBI  rather  than  text.  Example  statements  to 
be  rated  on  a  scale  from  strongly  agree  to  strongly  disagree 
included  responses  to:  "1  believe  I  had  sufficient  background 
knowledge  to  prepare  me  for  this  course;"  "The  CBI  portion 
enhanced  my  understanding  of  the  paper-based  course  material 
(student  guide,  FMs,  etc.);"  "The  computer  set  up  and 
administrative  procedures  were  too  much  trouble.". 

In  addition,  and  more  importantly,  the  test  officers  were 
asked  to  comment  on  each  lesson  (text,  CBI,  doctrine,  and 
examinations) ,  each  subject,  and  the  student  guide,  as  well  as  to 
provide  overall  comments  and  suggestions  on  the  course (s)  of 
instruction.  The  most  prevalent  free  response  comment  was  the 
request  for  more  CBI.  Others  requested  "more  practical  exercises 
and  computer  based  instruction."  Some  commented  on  historical 
references  and  lessons  learned,  and  on  the  graphics  and  humor. 
There  was  positive  feedback  on  the  quiz  format,  and  the  use  of 
CBI  in  general. 

BETA  testing  of  the  final  course  of  instruction  was 
conducted  on  networked  hardware.  To  ensure  accomplishment  of 
program  objectives,  test  results  and  appropriate  recommendations 
were  reviewed  and  approved.  Tester  comments,  changes, 
corrections  and  suggestions  were  reviewed,  and  as  appropriate, 
were  included  in  the  final  course  materials  prior  to  final 
printing  of  the  text  or  cutting  of  the  CD  ROM  for  CBI.  BETA  test 
results  were  provided  for  each  course  completed  (siammary  at 
Appendix  E) . 
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Conclusions 


The  Brigade-Battle  Staff  Training  System  was  designed, 
developed,  and  tested  to  meet  the  specific  training  needs  of  the 
ARNG,  although  it  is  evident  that  there  may  be  direct 
applicability  to  the  AC.  The  BSTS,  currently  still  a  prototype 
model,  could  potentially  be  used  at  home,  in  learning  centers,  in 
the  armory  or  state  academies  for  individual  study,  or  in  a 
networked  configuration  in  operational  units.  Personnel  could 
also  use  the  BSTS  as  a  part  of  or  following  the  Basic  and/or 
Advance  Courses  prior  to  assignment  as  a  battle  staff  officer,  or 
as  a  supplement  to  existing  courses  and  professional  education 
programs.  Although  this  project  began  with  individual  training 
modules,  it  also  has  potential  to  be  extended  upward  to  from 
brigade  to  division.  The  applicable  successes  and  lessons  learned 
from  the  BDE-BSTS  as  well  as  the  parent  program,  SIMITAR,  will 
enhance  the  individual  training  and  performance  of  all  battle 
staffs  and  individual  soldiers. 
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Appendix  A 

LIST  OF  ACRONYMS 

AAR  -  After  Action  Review 

AC  -  Active  Component 

ADA  -  Air  Defense  Artillery 

ADCOORD  -  Air  Defense  Coordinator 

ARI  -  Army  Research  Institute 

A2C2  -  Army  Airspace  Coinmand  and  Control 

ARNG  -  Army  National  Guard 

ARPA  -  Advanced  Research  Projects  Agency 

AV  -  Audio  Visual 

BDE  -  Brigade 

BN  -  Battalion 

BSHB  -  Battle  Staff  Handbook 

BSTS  -  Battle  Staff  Training  System 

CALL  -  Center  for  Army  Lessons  Learned 

CASS  -  Command  and  Staff  Services  School 

CBI  -  Computer  Based  Instruction 

CHEMO  -  Chemical  Officer 

CME  -  Contractor  Military  Expert 

COTS  -  Commercial  Off  the  Shelf  Software 

CS  -  Combat  Support 

CSS  -  Combat  Service  Support 

EMMii  -  Environment  for  Multi-Media  Interactive  Instruction 
ENG  -  Engineer 

FE  -  Formative  Evaluation 
FEA  -  Front  End  Analysis 
FSO  -  Fire  Support  Officer 

IPB  -  Intelligence  Preparation  of  the  Battlefield 

JRTC  -  Joint  Readiness  Training  Center 

LAN  -  Local  Area  Network 

NTC  -  National  Training  Center 

OAC  -  Officer  Advance  Course 
OBC  -  Officer  Basic  Course 
OPS  -  Operations 
ORG  -  Organization 
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PC  -  Personal  Computer 
PE  -  Practical  Exercise 
PH  -  Phase 

POI  -  Program  of  Instruction 
PSYOP  -  Psychological  Operations 

QA  -  Quality  Assurance 

RC  -  Reserve  Component 

SIGO  -  Signal  Officer 

SIMBART  -  Simulation-Based  Mounted  Brigade  Training  Program 
SIMITAR  -  Simulation  in  Training  for  Advanced  Readiness 
SIMUTA  -  Simulation  Based  Multi-Echelon  Training  for  Armor 
Units 

SME  -  Subject  Matter  Expert 

TCS  -  Tasks,  Conditions  LAN  -  Local  Area  Network,  and 
Standards 

TGT  POP  -  Target  Population 

TIS  -  Training  Information  System 

TRADOC  -  Training  and  Doctrine  Command 

WAN  -  Wide  Area  Network 
WFW  -  Windows  for  Workgroups 

XO  -  Executive  Officer 
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Appendix  B 

HARDWARE  AND  SOFTWARE  CONFIGURATION 

As  a  part  of  this  research  effort,  hardware  and  software 
specifications,  compatible  with  the  Battalion  and  Brigade-BSTS 
systems,  are  provided  to  identify  the  computer  stations  and  the 
training  management  system  used.  Specifications  were  finalized 
to  permit  procurement  of  compatible  systems  under  the  ARPA 
program.  Requirements  include  warranty,  technical  support,  and 
on  site  maintenance  support. 

Student  Station 

Hardware: 

486  DX2  Computer;  66  Mhz  processor 

8-16  Mb  RAM,  expandable  to  32+Mb;  70ns 

210-500  Mb  Hard  Disk 

16  bit  local  bus 

101  Key  Enhanced  Keyboard 

Desk  top  cabinet 

9600bps  modem,  or  better,  with  lOBT  RJ45  connector  (WAN 

configuration)  OR  Ethernet  adapter  (LAN  configuration) 
Spider  Graphics  card,  or  equivalent,  with  driver  to  handle 
640x480,  256  color  and  14"  monitor 

Software ;  DOS  6.21;  Windows  for  Workgroups™  3.11;  Icon  Author™ 
Present  6.01;  Training  Management  System  (TMS)/EMMii™ 

Accessories :  Surge  Strip;  10BT/RJ45  modular  plug  with  10'  cable; 
Mouse  Pad 


Manager  PC 

33Mhz,  or  faster,  486  PC,  16Mb  RAM 
200Mb+60Mb  per  subject,  disk  storage 

9600bps  modem,  or  better,  with  lOBT  RJ45  connector  (WAN 

configuration)  OR  Ethernet  adapter  (LAN  configuration) 

2  Shiva  net  modems  (WAN  configuration) 

Software:  DOS  6.21;  Windows  for  Workgroups™  3.11;  Icon  Author™ 
6.01;  Training  Management  System  (TMS)/EMMii™  TIS 

Accessories :  Surge  Strip;  10BT/RJ45  modular  plug  with  10'  cable; 
Mouse  Pad 
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Author  Station 


66Mhz,  or  faster,  486  PC,  16Mb  RAM 
Action  Media  II  (w/capture) 

Sound  Blaster  16  (w/ speakers) 

500  Mb  disk  storage 

9600bps  modem,  or  better,  with  lOBT  RJ45  connector  (WAN 

Ethernet  Adapter  (LAN  configuration 
150K/300K  double  spin  CD-ROM  or  equivalent 

Spider  Graphics  card,  or  equivalent,  with  driver  to  handle 
640x480,  256  color  and  14"  monitor 
2Gb  Tape  Drive 


) 


|.oftware:  DOS  6.21;  Windows  for  Workgroups™  3.11;  Icon  Author™ 
6.01;  Training  Management  System  (TMS) /EMMii ™  TIS  and  Student- 
Power  Point  6.0  ' 


Accessories;  Surge  Strip;  10BT/RJ45  modular  plug  with  10’  cable- 
Mouse  Pad  ' 
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Appendix  C 


INTRODUCTION  TO  BDM'S  ENVIRONMENT  FOR  MULTIMEDIA 
INTERACTIVE  INSTRUCTION 


The  EMMii  Concept 


The  training  management  system  as  an  Environment  for  Multi- 
Media  Interactive  Instruction  (EMMii)  has  been  developed  by  BDM 
as  an  enabling  environment  for  multi-media  computer  based 
training  (CBT)  development  and  delivery.  The  goals  of  EMMii  are: 

To  provide  a  state-of-the-art  environment  where  trainers 
can  develop  computer-based  lessons  for  students,  incor¬ 
porating  high  quality  still  photos,  full  motion  video, 
and  audio  in  the  lessons. 

To  assist  the  Training  Staff  in  the  management  of  their 
curriculum  and  materials. 

To  automate  the  administration  of  training. 

To  allow  students  to  proceed  at  their  own  pace  and  to 
refresh  themselves  as  necessary. 

To  provide  on-line  testing  in  the  subjects. 

To  record  student/ lesson  performance  data  in  an 
accessible  form,  and 

To  acquire  data  about  the  lessons  themselves. 

EMMii  uses  commercial  off-the-shelf  (COTS)  software 
integrated  into  a  system  that  allows  trainers  and  administrators 
to  make  good  use  of  the  specific  functionality  of  the  packages. 
The  Microsoft  Windows  for  Workgroups™  environment  then  provides 
a  familiar  look  and  feel  across  the  packages. 


C-1 


EMMii  Hardware  and  Software 


The  original  EMMii  was  developed  for  a  486  PC  platform  under 
05/2’’“.  However,  BDM  has  ported  this  system  to  a  Windows™ 
environment.  The  hardware  and  software  discussed  here  are  for 
the  Windows™  platform , 


Within  the  School,  Armory,  Learnitw 


WAN 


STANDALONE 


Figure  C-1 .  EMMii  hardware  architecture 

The  hardware  architecture  for  this  system  is  illustrated  in 
Figure  C-1.  EMMii  computers  are  all  486  PCs,  attached  through  a 
local  area  network  (LAN)  and/or  a  wide  area  network  (WAN)  using 
the  computer's  internal  or  external  modem,  ^nd  communicating  via 
Windows  for  Workgroups™  (WFW) .  The  lessons  are  stored  on 
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All  of  the  functional  areas  are  interrelated  and  communicate 
with  each  other  via  messages,  files,  and  database  queries. 

Figure  C-3  illustrates  the  interactions  between  the  Authoring 
Station,  the  Storage  Manager  (a  component  of  the  Manager  PC) ,  the 
TIS,  the  Instructor,  and  the  Student  Station. 


RING  STATION 


Figure  C-3 .  Interactions  of  EMMii  functional  areas . 

An  author  can  get  any  previously  created  files  and  templates 
from  the  Manager  while  on  his  PC.  Lessons  are  created  using  the 
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multi-media  authoring  package  IconAuthor (TM) .  An  author  can 
actually  work  on  a  lesson  locally  without  manager  interaction 
once  all  the  necessary  files  are  downloaded  to  the  authoring 
station.  When  the  lesson  is  completed,  procedures  on  the 
authoring  station  are  used  to  check  the  lesson  back  onto  the 
manager  station  where  the  lesson  and  its  associated  files  are 
stored. 

The  TIS,  which  has  knowledge  of  the  lessons  and  their 
suggested  order  within  the  course,  allows  an  instructor  to 
schedule  students  for  particular  lessons  and  at  specific  student 
stations.  The  storage  manager  and  the  student  station  then 
manage  the  download  of  the  lesson  files  and  the  students '  own 
data  (both  schedule  and  bookmark)  to  the  student  stations. 

When  a  student  takes  a  lesson,  the  results  (exam  results, 
questions  from  pose  a  question,  and  any  opinion  comments)  and 
bookmark  (where  the  student  ended  within  a  lesson)  are  collected 
at  the  prescribed  student  station.  Upon  completion  of  a 
student's  session,  the  results  are  transmitted  to  the  database. 
If  no  other  students  are  scheduled  for  the  same  station/lesson 
within  24  hours,  the  lesson  is  automatically  deleted  from  the 
student  station  upon  completion. 

EMMii  Course  Structure 


The  highest  level  of  organization  within  EMMii  is  a  course, 
A  course  consists  of  subjects,  and  the  subjects  are  made  up  of 
one  or  more  lessons.  The  organization  is  illustrated  in  Figure 
C-4. 


Courseware  and  Associated  Multi-media  Files 


Figure  C-4.  EMMii  course  structure. 
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Normally  only  one  author  will  work  on  a  lesson.  The  lesson 
consists  of  a  multimedia  presentation  produced  by  an  author  as 
well  as  text  files,  audio  files,  stills,  motion  video,  and 
graphics  files  used  in  the  presentation.  All  of  these  files,  as 
shown  in  Figure  C-5,  become  associated  with  the  lesson  and  are 
stored  on  the  manager  PC  after  the  lesson  is  checked  in  through 
the  authoring  software. 


CBI  LESSON 


SCRIPTS  (ICONWARE  AND  PATHS) 

AUDIO  (WAVE  HLES) 

MOVIES  (AVI  FILES) 

GRAPHICS  (BMP  AND  PCX) 

TEXT  (USER  FILES  AND  lA  “SMART  TEXT’) 


ANIMATION  (SCRIPTS) 


VARIABLES  (TEXT  FILES  FOR  USE  BY  SCRIPTS) 


Figure  C-5.  An  example  lesson  and  associated  files. 

The  files  are  downloaded  to  users  (authors,  instructors,  and 
students)  whenever  the  lesson  is  referenced.  The  display  type 
and  suffix  of  the  files  are  maintained  by  the  storage  manager  to 
allow  identification  at  the  other  levels  when  downloaded. 

Maintenance  of  files  and' versions  of  files  are  performed  by 
the  storage  manager  (on  the  manager  PC) .  Common  files  may  be 
shared  by  lessons  within  a  subject,  by  subjects  within  a  course, 
or  by  courses  within  the  entire  system. 

EMMii  Functionality 

The  following  provide  somewhat  more  detailed  descriptions 
of  the  EMMii  functions  than  were  provided  in  Figure  C-1. 

Author  Station  Software 


Most  of  EMMii  is  interactive.  Figure  C-6  shows  a  typical 
EMMii  screen.  This  particular  screen  is  used  by  authors  to  enter 
short  descriptions  of  their  files,  in  order  to  make  browsing 
easier  when  the  lesson  is  stored. 


C-6 


Project  File  Description 


Pftscri&Bom  SO  Chaiiftcter*  Maximum 

Questions/Ansrs  on  E-Mail 


BACKGRND.  BMP 

BOOKMARK.IW 

BdOKMARK.VAR 

CHAPTERS.VAR 

CHKBKMK.IW 


^ I  Sort  by 


CuirentProiect 


O  File  Name 


File  Type 


^dAWFWpro|ctsoi>tn^ 


j  Select  Project  { 


I  uispiay  nequirea  riiesj  ;  i  Mwept  t  v  i  t  \  neip  r  i  Ulv 


Figure  C-6.  Typical  EMMii  screen 

In  addition  to  the  scripting  functions  provided  by  the 
IconAuthor™  package,  the  EMMii  author  station  software  provides 
the  following  specific  functions: 

Temporary  storage  and  retrieval  of  lessons 

Fetching  files  from  previously  stored  lessons 

Checking  common  file  usage 

Entering  descriptions  of  lesson  files 

Checking  lessons  in  and  out  of  the  storage  manager 

Creation  of  a  new  lesson 

An  author  may  also  want  to  use  the  TIS  to  look  at  course 
organization  as  well  as  student  results.  The  TIS  can  be  accessed 
through  the  author  administration  functions . 

Along  with  the  EMMii  author  software,  special  IconAuthor 
templates  and  subroutines  have  been  developed  to  allow  authors  to 
retrieve  and  use  data  provided  by  EMMii  (such  as  the  bookmark, 
course/  subject/lesson  names)  and  to  provide  data  back  to  EMMii 
upon  completion  (scores,  answers,  updated  bookmark,  student 
assessments) .  Other  special  templates  have  been  developed  to 
"automate"  some  standard  procedures  (use  of  the  bookmark,  certain 
test-types,  et  al.)  In  addition,  BDM,  in  concert  with  subject 
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matter  experts  and  training  developers,  can  develop  custom 
templates  and  shells  to  be  used  throughout  the  organization's 
courseware,  to  give  it  a  standard  look  and  feel. 

Instructor  Station  Software 


Instructors  perform  their  work  through  the  TIS.  The  TIS  can 
be  accessed  through  both  the  author  administration  and  the 
instructor  PCs.  The  TIS  allows  an  instructor  to  schedule  student 
sessions,  view  results,  and  look  at  course  structure. 

The  other  TIS  functions  used  by  an  instructor  include: 

-  Querying  the  status  of  a  student  station 

Adding  and  deleting  permanent  lessons  from  student 
stations 

-  Aborting  a  student's  session 
Student  Station  Software 


The  student  station  software  consists  of  one  log-in  screen 
which  is  always  displayed  when  the  station  is  turned  on  (booted 
up).  In  addition,  it  contains  several  background  processes  that: 

-  Receive  "schedule"  messages  from  the  database  and 
initiate  lesson  downloads 

Ensure  that  students  receive  the  lessons  for  which  they 
are  scheduled 

Retrieve  student  test  results  and  bookmarks  and  log  them 
to  the  "Host"  computer  (either  the  Manager  Computer  or 
the  Instructor  Station) 

Clean  up  the  student ' s  results  and  course  materials  after 
completion. 

The  Training  Information  System  (TIS) 

■The  'TIS  serves  several  purposes  within  EMMii.  Its  primary 
function  is  to  provide  course  administrators  and  instructors  with 
information  about  courses  that  are  available  and  information 
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about  student  training  activities .  The  TIS  also  maintains 
detailed  records  on  student  training  performance  and  on  student 
assessments  of  the  training  received.  In  addition,  the  TIS 
assists  the  instructor's  scheduling  of  students  for  particular 
lessons  at  specific  stations  for  specific  times;  thereby  avoiding 
conflicts  with  student  time  and  station  scheduling,  maximizing 
utilization  of  stations,  and  helping  to  assure  that  students 
receive  training  in  the  proper  sec[uence.  Figure  C-7  shows  a 
typical  EMMii  TIS  screen,  this  one  reporting  on  the  students  who 
have  taken  a  particular  lesson. 


ESI 


Training  Info  System  [Lesson  History] 


fcii 


Form  Edit  Records 


Lesson  History 

tlS  BBBB  Lsiggie..! 


Help 


J 


1 


c 

uommon  Core  t-  f 

=3 

c 
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r 
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Figure  C-7 .  Typical  EMMii  TIS  screen 

EMMii  uses  the  Microsoft  Access™  relational  database 
management  system  software.  Using  Access  provides  a  similar  look 
and  feel  to  the  other  EMMii  functions.  The  TIS  consists  of  over 
50  screens  and  reports  that  allow  a  user  to: 

Add/delete  students 

Organize  curricula 

Schedule  student  training  sessions 

View  training  status  by  student  or  by  lesson 

View  exam  results  by  student  or  by  lesson 

Archive  course  materials 
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The  Storage  Manager 


The  purpose  of  the  EMMii  storage  management  module  is  to 
provide  a  transparent  disk  storage  system  where  users  on  any 
station  are.  unaware  of  the  actual  storage  structure  used  for 
EMMii  lesson  files.  The  storage  manager  provides 

an  efficient  storage  and  retrieval  system  for 
individual  EMMii  lesson  files 

an  efficient  system  for  transferring  complete 
lessons  between  PC  workstations  and  the  manager  system 

Given  the  probability  that  the  lessons  developed  will 
undergo  changes  with  time,  configuration  management  of  the 
courseware  is  performed  by  the  storage  manager.  The  software 
also  allows  designation  of  files  as  common  to  several  lessons, 
decreasing  storage  space,  and  making  global  changes  easier  to 
perform. 

Communications 


The  EMMii  communications  software  resides  on  each  PC  in  the 
system.  It  uses  Windows  for  Workgroups  to  perform  message  and 
file  transmission  over  an  existing  LAN  or  WAN  through  the  use  of 
a  modem  and  standard  telephone  lines.  In  addition,  small  local 
databases  are  used  by  EMMii  software  to  allow  machines  in  the 
system  to  access  data  about  what  is  happening  on  another  PC  (such 
as  ins true tor /student  station  interactions). 


C-10 


Appendix  D 


DEVELOPMENT  OF  COMPUTER  BASED  INSTRUCTION 


Introduction 


Early  in  this  program,  BDM  developed  a  process  for  setting 
standards  and  completed  two  iterations  of  meetings  to  develop 
standards  for  the  Computer  Based  Instruction  (CBI)  used  in  both 
the  Battalion  Battle  Staff  Training  System  (BN-BSTS)  and  the 
Foin^ard  Support  Battalion  (Combat  Service  Support)  Training 
System  (CSS-BSTS)  for  the  Army  National  Guard. 

Since  the  initial  development,  the  standards  have  continued 
to  be  refined.  This  report  explains  the  standards  used  by  BDM  in 
developing  the  Battle  Staff  Training  System.  There  are  also 
references  to  the  data  collected  and  sent  to  the  armory  via  the 
training  management  system,  EMMii  -  Environment  for  Multi-Media 
Interactive  Instruction. 

The  areas  addressed  for  CBI  Delivery  Standards  are  in  Figure 
D-1.  Descriptions  and  details  of  the  components  of  each  standard 
are  contained  below. 


INSTRUCTION 
ASSESSMENT 
LOOK  AND  FEEL 
FUNCTIONS 
MANAGEMENT 


Figure  D-1.  CBI  delivery  standards. 


Instruction 


.0^3 gctives .  The  enabling  knowledge  for  entrance  into  the 
t  literacy  with  computers  and  completion  of 

Course  (OBC)  and  Officers  Advanced  Course 
inS  in  respective  branch  officer  for  all  courses  except  S3 

completion  of  Command  and  Staff  Services  School 
(CAS3)  and  Command  and  General  Staff  School  is  recommended. 

required  to  be  branch  qualified  for 
de  (e.g.,  served  as  a  platoon  leader,  executive  officer 
or  specialty  platoon  leader  or  assistant  staff  officer  for  a 

position  (SI,  S2,  S3,  or  S4)  ,  company  oommander  and 
primary  staff  officer  for  S3  and  XO) 

Prior  to  beginning  a  specific  staff  officer  functional 

battle  staff  will  complete  the  lessons 

in  the  Coinmon  Core  Course. 

levels  of  learning  for  a  lesson  include  Level  1  - 
a  skills  (individual  definitions  and  components  of  the 
functional  area) ,  Level  2  -  individual  problem  solving 
(g  a  problem,  apply  knowledge,  produce  a  behavior),  and  Level 

basic  skills,  production  of  individual  staff 
output,  and  practical  exercises.  (Collective  tasks  may  be 

^  future  program  to  train  dyads  and  triads  (small 
STMRART^^^  then  full  staff  training  through  the  SIMUTA  and 

iy®^T  programs.)  The  form  of  training  objectives  is  Task  - 
Condition  -  Standard. 


o^^ganizational  competency  model  follows  Olmstead's  model 
Srget^Iudi2!ce°^°^^  modified  to  better  meet  the  needs  of  the 

The  instruction  allows  students  to  take  the  subject  "for 

which  case  test  results  are  recorded.  However,  if  the 
chooses  to  take  a  subject  for  reference,  merely  to  peruse 
a  subject,  some  tests  will  not  be  made  available. 

Scenario  and  METT-T.  Common  scenarios,  based  on  the  NTC  and 
synchronized  with  the  SIMBART,  SIMUTA  and  CSS  programs,  drive  the 
basic  skills  and  problem  solving  stages  of  instruction.  The 
common  scenario  normally  drives  levels  1  and  2;  or  scenarios 

individual  training  objective  are  adapted  for 
stages  1  and  2 ,  when  required.  The  factors  of  enemy,  troops, 
and  terrain  (and  weather)  are  stabilized  for  training  in  all 
defense  factor  of  mission  is  varied  for  offense  and 
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Course  Structure  and  Mapping.  The  highest  level  of 
structure  is  the  course  (SI,  S2,  S3,  ...  SIGO) ,  Courses  are  10-50 

hours  duration.  The  next  (lower)  level  of  structure  is  the 
subject.  A  subject  is  a  grouping  of  lessons.  (See  Figure  D-2 . ) 

The  subject  is  a  piece  of  IconAuthor  (tm)  courseware  and 
associated  multimedia  files.  Most  subjects  are  of  1-2  hours 
duration  (in  the  CBI) .  Within  the  lesson  there  are  topics. 


Courseware  and  Associated  Multi-media  Files 


Figure  D-2 .  General  EMMii  curriculum  structure 

Each  course  is  a  mixture  of  text -based  material  and  CBI. 

All  criterion  referenced  assessment,  performance  measurement  and 
diagnostic  testing  is  done  within  the  CBI.  There  is  a  short  (5-10 
minute)  introductory  CBI  subject  for  each  course  and  lesson  as 
well  as  a  pretest  (see  testing,  page  D-8) .  The  lessons  themselves 
are  available  through  a  menu  in  the  CBI.  The  introduction 
stresses  the  importance  of  the  material  to  the  officer's  job  and 
how  the  particular  material  fits  into  the  overall  curriculum.  It 
also  indicates  what  doctrinal  reading  should  be  done  in 
conjunction  with  the  lesson  and  prior  to  taking  the  exam.  For 
some  lessons,  the  introduction  and  exam  are  the  only  components 
of  the  lesson  within  the  CBI.  Other  lessons  include  additional 
CBI  material  (tutorials,  practical  exercises,  or  examples  from 
lessons  learned)  to  strengthen  the  learning.  Lessons  with  CBI  are 
indicated  by  a  check  mark  on  the  course  map.  The  course  map  is 
included  in  both  the  student  guide  and  CBI  (see  section  below) . 
There  is  more  on  testing  philosophy,  navigation,  and  structure  in 
later  sections. 
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Strategies .  The  specific  strategy  applied  to  achieve  a 
specific  training  objective  was  selected  by  the  development  teaim 
SMEs  in  conjunction  with  the  instructional  designer  following  the 
model  of  design  and  development  upon  which  each  team  member  was 
trained.  The  strategies  applied  in  the  course  development  process 
are:  tutorial,  drill  and  practice,  guided  discovery,  problem 
solving,  simulation,  games,  and  cooperative  learning.  Practice  is 
incorporated  to  the  maximum  extent  possible  to  attain  the 
training  objective. 

Training  for  the  development  team  included  the  best 
application  of  a  strategy  to  a  specific  training  objective.  The 
SME  development  team  was  also  trained  in  behaviorally-oriented 
principles  of  instruction  which  are  appropriate  to  consider  in 
designing  specific  instructional  strategies. 

Types  of  Interaction.  For  each  strategy  selected,  the  SME 
has  designed  the  appropriate  type  of  interaction  to  best  achieve 
the  training  objective.  Types  of  interaction  include  recognition, 
recall,  comprehension,  application/analysis,  synthesis,  and 
evaluation.  (See  also  section  on  feedback,  diagnostics,  and 
testing,  below.) 

Role  of  the  Mentor.  Clearly,  the  optimiam  method  to  achieve 
a  high  performing  staff  member  and  battle  staff,  particularly  in 
high  order  cognitive  tasks,  is  through  the  availability  of  a 
mentor  during  the  process .  An  instructor  or  trainer  is  strongly 
recommended  and  can  be  made  available  through  the  E-Mail  and  Chat 
functions  provided  through  the  EMMii  and  Windows  for  Workgroups 
(WFW) .  (See  section  on  support  systems,  below.)  At  any  time 
students  have  the  ability  to  pose  a  question  that  will  ultimately 
be  posted  for  response  by  a  mentor,  instructor,  or  trainer.  As 
the  battle  staff  progresses  through  the  course  structure  (stages 
I  -  III) ,  there  may  be  an  increased  need  for  the  mentor, 
instructor,  or  trainer. 

Motivation .  Techniques  applied  by  the  SME/design  team  to 
motivate  the  student  (and  overcome  the  commonly  observed 
attrition  effect  in  distance  learning)  include  hxamor,  fantasy, 
games,  challenges,  and  competition  with  self  and/or  others.  These 
techniques  are  applied  based  on  the  learning  stage,  type  of  skill 
and  instructional  method. 
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Principles  for  Media  Use.  Use  of  media  is  tailored  to  the 
training  objective  and  to  meet  the  test  of  effectiveness 
(training  and  cost) .  Video  is  used  judiciously  for  key  points, 
to  gain  attention,  or  for  motivation.  It  is  applied  for 
demonstration  of  principles,  equipment,  lessons  learned,  to 
provide  a  historical  perspective,  and,  to  a  minor  extent,  for 
variety. 

Most  screens  use  audio  to  emphasize  key  points  and  supply 
additional  information  to  the  learning.  A  male  voice  is  used  for 
the  target  population.  Existing  (TASC)  audios  or  videos  have 
been  used  in  some  cases;  however,  most  have  been  developed  by  the 
SME,  and  produced  by  the  CBI  team,  specifically  to  support  the 
learning  objective. 

Text,  at  a  10th  grade  reading  level,  is  used  on  most  every 
screen  and  amplified  with  audio,  video,  graphics,  photo  stills, 
or  animation.  Animation  is  used  judiciously  for  motivation  or 
variety. 

Graphics  and  photo  stills  are  used  to  anchor  a  learning 
objective  and  apply  doctrinal  publications  (Field  Manuals, 

ARTEPS,  and  Tactics,  Techniques  and  Procedures)  to  ensure  the 
officer  can  quickly  refer  to  the  doctrinal  publication  in  field 
application. 

Assessment 


Diagnostics  and  Prescription.  Each  subject  area  begins  with 
a  diagnostic  test  (pretest)  that  is  identical  to  the  final 
subject  examination.  The  student  is  not  given  the  correct  answers 
to  the  pretest  questions.  However,  he  is  shown  which  questions 
he  missed  and  is  told  (on  the  main  menu)  the  lessons  he  did  well 
enough  on  to  skip  over  during  the  lessons  for  the  subject.  If  the 
student  desires,  he  has  the  opportunity  to  complete  lessons  on 
which  he  has,  through  the  diagnostic  pretest,  demonstrated 
mastery. 

Testing.  There  are  three  types  of  tests:  quizzes,  lesson 
exams,  and  pretests/final  exams.  Quizzes  are  graded  to  provide 
the  student  a  measure  of  his  performance;  but  results  are  not 
stored  in  the  database  for  future  analysis.  Exams  (lesson  and 
final)  and  pretests  are  graded,  issued  a  percentage  score, 
identified  as  to  pass/fail,  and  stored  for  analysis  and  review. 

A  student  may  take  any  quiz,  lesson  exam,  or  final  as  many  times 
as  he  wishes. 
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Each  subject  begins  with  a  pretest  that  the  student  must 
take  before  taking  the  subject  for  credit.  The  student  may  take 
this  test  only  once.  The  EMMii  tracks  that  the  pretest  has  been 
taken  and  does  not  permit  it  to  be  re- taken.  If  the  student 
passes  the  pretest  (achieves  a  score  of  80%  or  better  on  each 
lesson  in  the  subject) ,  he  is  considered  to  have  passed,  and  can 
either  proceed  to  another  subject,  or  continue  with  the  one  at 
hand.  Students  may  always  return  to  a  subject  they  have  taken 
previously  for  review  or  additional  study. 

There  is  also  a  final  mastery  exam  which  covers  the  entire 
subject.  The  final  is  identical  to  the  pretest  (except  that  the 
questions  are  re-ordered  and  feedback  is  given  on  the  correct 
answer  and  where  the  student  can  gain  the  information  to  learn 
the  objective  measured  by  the  question.)  A  minimiom  score  of  80% 
is  required  for  mastery  before  the  student  is  considered  to  have 
"passed/receive  a  GO"  on  the  subject. 

The  examination  format  is  multiple  choice  or  a  derivative 
thereof  in  most  cases,  as  the  computer  cannot  (easily  or 
accurately)  grade  subjective  type  examinations.  Derivatives  of 
the  multiple  choice  include  matching,  true-false,  sorting,  choose 
from  a  list,  choose  spots  or  locations  on  a  map,  or  graphic,  or 
use  of  f ill-in- the-blank. 

Within  each  subject,  each  lesson  has  an  associated  lesson 
examination.  This  exam  is  graded  and  recorded  in  the  data  base.  A 
minimum  score  of  80%  is  required  for  mastery.  Mastery  of  the 
material  is  required  before  the  CBI  will  advise  the  student  to 
move  to  the  next  lesson  or  the  exam.  Most  lessons  also  contain 
quizzes  or  practical  exercises  (PEs) .  The  PEs  may  or  may  not  have 
"correct"  answers  associated  with  them.  Most  have  only 
"suggested"  answers  designed  to  encourage  the  learning  of  the 
cognitive  concept  or  model  behind  the  PE  for  application  under 
different  factors  of  METT-T.  Responses  to  questions  in  PEs  are 
neither  graded  nor  recorded,  to  encourage  the  battle  staff 
officer  to  take  risks,  try  new  concepts  and  enhance  the  cognitive 
learning  process. 

Feedback  on  Test  Performance  and  Remediation.  As  mentioned 
in  the  section  on  testing,  there  are  three  types  of  tests  used. 
The  answers  and  scores  on  two  of  the  three  types  are  recorded 
within  the  EMMii.  The  following  addresses  user  feedback  on  each 
type . 
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Quizzes  and  exercises  are  designed  to  instruct  the  user  in 
terms  of  what  has  been  learned  or  still  needs  work.  Immediate 
(right  or  wrong  for  quizzes)  feedback  is  given  on  each  question, 
with  reinforcement  for  right  answers,  and  inferences  as  to  what 
is  correct  for  wrong  answers.  In  some  cases  the  question  is 
re-asked  when  the  answer  is  wrong.  The  PEs  also  apply  this 
feedback  concept. 

Fill-in-the-blank  exercises  are  used  sometimes.  The  computer 
grades  these  by  comparing  the  user's  response  to  a  list  of 
acceptable  responses . 

Exams  are  not  used  as  a  GO/NO  GO,  or  gate,  between  lessons 
but  merely  as  a  suggestion  for  student  progression.  Failure  on  a 
lesson  exam  (not  meeting  the  80%  threshold)  does  not  prevent  the 
student  from  proceeding  to  another  lesson.  On  the  lesson  exams, 
the  user  is  provided  specific  feedback  for  each  question  through 
the  correct  response  and  additional  information  to  anchor  the 
learning  objective. 

On  a  pretest,  the  user  is  shown  which  questions  were  missed 
and  told  the  lesson  (within  the  subject)  to  which  the  missed 
question  applies.  Based  on  the  areas  missed,  the  user  is  shown 
the  main  menu  with  thumbs  up/down  to  indicate  which  lessons 
should  be  re-examined  before  taking/retaking  the  final  exam. 

Similarly,  upon  completion  of  the  final  exam,  the  students 
are  shown  which  lessons  they  did  well  on.  Individual  feedback 
(right /wrong  and  an  explanation)  is  given  for  each  question. 
Normally  the  correct  answer  for  these  questions  is  shown, 
although  in  some  cases,  the  specific  feedback  indicates  the 
correct  answer.  Additionally,  feedback  will  refer  the  student  to 
text/CBI  for  study  to  achieve  the  learning  objective. 

Look  and  Feel 


The  following  standards  reflect  what  is  readily  available 
within  the  Windows  and  the  IconAuthor  environment.  Templates 
were  created  to  make  much  of  the  navigation  and  functionality 
more  uniform  across  lessons,  subjects  and  courses  and  enhance  the 
time  required  to  produce  the  CBI. 

Colors .  Menus  and  opening  screens  have  a  dark  gray  or  blue 
background  with  silver  buttons.  Individual  screens  within 
lessons,  introductions,  exercises,  etc.,  use  a  variety  of  colors 
to  enhance  interest,  readability,  and  maintain  student  attention. 
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Text  is  in  a  variety  of  colors .  Menus  and  opening  screens 
use  black,  white,  silver,  navy,  and  maroon  (for  emphasis) .  Other 
screens  use  colors  that  provide  good  contrast  to  the  background. 
Use  of  florescent  colors  has,  normally,  been  avoided  because 
these  often  appear  to  pulsate  on  a  bright  screen  and  can  thus  be 
annoying , 

Fonts .  Arial  is  used  for  most  text;  12  point  and  14  point 
are  preferred  for  normal  text,  with  larger  sizes  for  titles.  In 
some  instances  10  point  arial  is  used  when  there  is  inadequate 
space  for  a  larger  font.  Buttons  are  in  a  standardized  font 
across  the  entire  system  (all  courses,  subjects,  and  lessons)  as 
they  are  based  on  templates  designed  for  the  entire  system. 

Button  Placement.  Navigational  buttons  at  the  bottom  of  the 
screen  are  consistent  throughout  all  the  courseware.  For  PEs, 
buttons  are  placed  in  the  logical  location  to  aid  student 
learning. 

Borders .  Screens  do  not  have  borders  on  them.  This  enabled 
us  to  make  full  use  of  the  space  available.  Many  screens  will 
have  a  small  text  field  or  icon  in  the  upper  left  corner  as  an 
indicator  of  the  subject  to  aid  the  student  in  navigation. 

Screen  Layout.  Menus  contain  no  more  than  eight  menu 
buttons,  with  screen  navigation  buttons  at  the  bottom.  They  are 
generally  240  by  40  pixels,  larger  if  needed. 

Menu  buttons  generally  correspond  to  lessons  or  topics. 
Normally  the  user  will  progress  through  the  lessons /topics  in  the 
sequence  designed  by  the  SME  and  as  shown  (top  to  bottom) . 

Figure  D-3  provides  the  standard  template  for  a  subject  main 
menu.  To  aid  the  user  in  navigation,  the  main  menu  is  titled 
with  the  course  and  subject,  designated  in  Figure  D-3  as  @  course 
and  @  subject. 
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@SubJeict 


Figure  D-3 .  Sample  main  subject  menu 

Submenus  are  used  to  allow  the  user  access  to  individual 
items  within  a  lesson.  These  include,  at  a  minimum,  the  lesson 
introduction  and  lesson  exam.  In  some  instances,  they  provide 
access  to  topic  material  and/or  practical  exercises.  In  either 
case,  the  same  "look  and  feel"  standards  mentioned  above  apply. 
The  submenu  contains  the  name  of  the  course  and  lesson  instead  of 
the  subject  and  lesson. 

Other  than  menus,  most  other  material  is  shown  on  a  screen 
such  as  Figure  D-4  which  shows  a  student  who  has  passed  the 
lesson  exam.  Buttons  are  40x160  (Options  and  Replay)  and  40x120 
(Next  »  and  «  Previous) .  Not  all  buttons  are  available  all  the 
time.  Those  not  available  are  grayed. 

The  Next  >>  button  is  used  for  page  turning  when  needed. 

Not  all  screens  require  the  user  to  press  Next  »  to  get  to  the 
following  screen,  but  any  screen  with  substantial  amounts  of  text 
or  diagrams  allows  the  user  to  choose  when  to  proceed. 

The  «  Previous  button  is  the  reverse  of  Next  >>. 
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Figure  D-4 .  Sample  lesson  menu. 


The  result  of  ”«  Previous  "  is  not  necessarily  "the 
previous  screen" .  For  example,  one  may  show  a  diagram  with  audio, 
followed  immediately  by  a  text  screen  referring  to  the  diagram. 
The  text  screen  contains  the  Next  »  button,  which  sends  the  user 
to  a  third  screen.  The  "<<  Previous  "  button  on  the  third  screen 
might  more  naturally  send  the  user  back  to  the  diagram  rather 
than  the  text. 

The  Replay  button,  when  used,  is  more  immediate.  It  is  used 
to  replay  an  audio  or  an  animation.  Again,  Replay  may  cause  the 
user  to  replay  several  screens . 

The  Options  button  is  a  catch-all.  It  produces  a  sub-menu  of 
options.  These  normally  include  C2uit,  Menu,  Cancel,  Glossary,  and 
a  Pose  a  Question  function.  A  Cancel  button  removes  the  options 
from  the  screen.  Figure  D-5  provides  a  sample  of  what  these 
OPTIONS  look  like.  Again,  not  all  of  these  are  available  all  the 
time,  although  the  Glossary  and  Pose  a  Question  functions  are 
available  most  of  the  time.  The  Menu  function  returns  the  user 
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to  the  menu  on  which  the  current  item  (exam,  lesson,  topic, 
exercise,  etc.)  resides.  Other  standards  for  glossaries,  etc., 
are  discussed  below.  Other  accessories  under  OPTIONS  include 
Credit?,  Map,  and  Your  Opinion.  The  Glossary  and  Pose  a  Question 
functions  are  not  available  during  exams  or  the  pre-test  or  final 
exams . 


IjPpse  aiOu^stipn  < 


Figure  D-5.  Sample  options  menu 


In  the  case  of  a  multiple  choice  question,  50  by  50  pixel 
buttons,  labeled  A-E  are  used.  If  the  question  has  fewer  than 
five  possible  answers,  unneeded  buttons  are  not  shown. 

For  most  questions,  the  Next  >>  button  is  used  by  the  user 
to  register  a  response,  thus  allowing  the  student  to  change  the 
answer  before  it  is  recorded  and/or  checked  in  to  the  data  base. 

Functions 


Navigation.  Initially,  the  user  is  shown  the  screen  in 
Figure  D-3 .  If  the  student  has  never  taken  the  pretest  for  this 
subject,  the  pretest  button  is  available  (ungrayed)  and  the  final 
exam  button  is  unavailable  (grayed) .  If  the  pretest  has  been 
taken  previously,  the  pretest  button  is  grayed  and  the  final  exam 
button  is  ungrayed.  In  either  case,  the  student  may  look  at  the 
introduction  or  go  to  the  main  menu.  If  the  main  menu  is  chosen, 
the  student  is  asked  if  he  wants  to  take  the  subject  for  credit. 

Whenever  a  student  exits  the  CBI,  a  "bookmark"  is  placed 
within  the  local  database  and  eventually  the  host/Armory 
database.  The  next  time  the  student  returns  to  this  subject,  the 
main  menu  will  look  the  way  it  did  prior  to  exit. 
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The  lesson  menu  will  indicate  with  a  thumbs  up  whether  or 
not  the  lesson  has  been  passed  through  the  lesson  exam,  the 
pretest,  or  the  final  exam.  If  the  lesson  has  not  yet  been 
passed,  the  item  on  the  menu  last  viewed  by  this  user  is 
italicized.  Figure  D-4  gives  an  example  of  this.  Once  the  user 
has  taken  the  final  exam  at  least  once,  the  main  menu  will 
indicate  (via  thiimbs  up)  which  lessons  were  passed  on  the  final. 


Glossaries  and  accessories .  Each  course  has  a  glossary 
designed  by  the  SMEs  to  aid  learning.  The  glossary  is  available 
through  the  Options  button.  The  Glossary  itself  is  a  Windows  Help 
File,  and  as  such  has  hot  spots  and  keyword  searches  available. 
The  first  page  of  a  glossary  is  shown  in  Figure  D-6.  The  user  may 
return  to  the  last  screen  he  was  on  by  using  the  Previous  button. 


Battle  Staff  Glossary 


Eile  Edit  Bookmark  Help 
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area  of  interest  (AI)  That  area  of  concern  to  the  commander,  including  tfie  area 
areas  adjacent  thereto,  and  extending  into  enemy  territory  to  the  objectives  of  curt 
operatLons.  This  area  also  includes  areas  occv^ied  by  enemy  forces  who  could  je 
accomplishment  of  the  mission. 

(CBI:  S2/1A2;  X0/^A2) 


Figure  D-6.  Battle  Staff  Glossary 
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other  accessories  include  a  Course  Map  and  the  Pose  a 
Question  function  on  the  Options  key.  These  are  discussed  below 
and  in  the  section  on  management. 

The  student  is  allowed  to  type  in  a  question  for  the 
trainer /mentor  by  the  Pose  a  Question  function  in  the  Options. 
The  question,  plus  some  information  about  where  in  the  subject 
the  student  was  when  the  question  was  asked,  are  written  to  a 
file.  This  f ile , is  automatically  sent  to  the  host  computer 
(armory)  system  when  the  student  uploads  results.  (See  support 
systems,  below.) 

Management 


Data  Collected  from  Training  Sessions .  Every  exam  within  a 
lesson  has  a  number.  This  number  along  with  the  type  of  exam 
(lesson,  pretest  or  final  exam)  is  recorded  in  the  database,  as 
are  all  the  user's  responses  to  computer-graded  questions.  Thus, 
for  example  EMMii  has  access  to  what  answer  (A-E)  the  student 
gave  to  question  Number  3  on  Exam  2 .  If  the  student  takes  an  exam 
more  than  once,  each  iteration  is  kept  as  a  separate  record. 

The  start/stop  date/times  for  each  student  session  and  the 
student's  bookmark  are  recorded  in  the  database.  Other  data  are 
also  maintained: 

Names  and  social  security  numbers . 

Current  progress  in  courseware. 

Courses  being  used. 

Course  (s)  curriculum. 

Data  Format.  Attached  are  sample  screens  and  reports  from 
the  EMMii  database.  These  provide  an  idea  of  the  type  of  data 
available.  Additional  data  are  available  in  the  EMMii  user's 
guide . 

The  EMMii  does  allow  the  database  administrator  to  establish 
roles  that  control  access  for  users .  These  roles  then  determine 
which  screens,  functions  and  reports  that  a  user  may  access. 

Control  over  Progression.  Originally,  it  was  expected  that 
the  EMMii  would  control  the  student's  progression  through  the 
courseware  by  disallowing  access  to  a  subject  if  its 
prerec[uisites  had  not  been  passed,  but  in  its  current  form  a 
student  can  access  any  subject  or  lesson  for  which  he  is 
scheduled  in  any  order.  The  student  will  be  told  which 
subject (s)  have  been  accessed  and/or  passed. 
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What  distinguishes  a  course  from  a  subject  from  the  user's 
viewpoint  will  be  the  fact  that  within  subjects  there  are  menus 
and  screens  that  allow  him  to  allow  navigation  between  lessons 
and  topics,  whereas  going  from  one  subject  to  another  will 
generally  involve  an  exit  from  the  specific  courseware  to  the 
EMMii  student  environment. 

The  standards  discussed  in  this  document  (size,  content,  and 
structure)  are  for  a  Subject.  Within  a  course,  the  student 
progresses  from  subject  to  subject.  Within  a  subject,  he  goes 
from  lesson  to  lesson. 

The  user  is  registered  at  the  host  computer  (armory)  for  a 
course  or  courses  and  is  provided  with  a  schedule  disk;  CD  ROM(s) 
containing  the  courseware;  the  student  guide  containing  the  text, 
curriculum,  job  aids,  maps,  operations  order,  instructions  for 
assembling  the  computer,  etc.;  and  a  student  computer.  The 
student  then  can  take  the  computer  home  and  study  at  his  own  pace 
and  sequence,  and  become  a  proficient  battle  staff  officer. 

Support  Systems.  Outside  the  courseware  is  the  EMMii 
student  environment .  The  user  has  several  functions  available 
here,  as  shown  in  Figure  D-7 . 

The  user  is  able  to  connect  to  the  host  computer  via  a 
(Shiva)  net  modem  located  at  the  armory  end  and  appropriate  modem 
software  at  the  student's  machine.  The  connection  is  performed 
easily  via  the  EMMii  Student  Station  software.  Once  connected, 
the  student  has  the  use  any  of  the  network  functions:  results 
upload.  E-mail,  and  chat.  Results  upload  sends  data  from  the 
student ' s  local  database  to  the  host  database  at  the  armory  for 
review  by  the  trainer.  Any  questions  which  were  posed  are  also 
sent  to  the  host  database.  The  E-mail  facility  is  the  standard 
one  provided  with  Windows  for  Workgroups  (WFW) .  It  allows  the 
user  to  send  mail  to  other  users  and  to  receive  mail.  The  chat 
facility  is  also  via  WFW.  It  allows  users  on  the  network  to 
converse  with  each  other  in  real  time  through  text  messages . 

The  user  is  also  able  to  find  out  where  in  the  course 
progression  he  stands  (lesson  summary)  and  what  he  should  take 
next.  This  is  available  through  the  EMMii  student  environment, 
not  the  courseware. 
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Figure  D-7 .  Student  functions  available  in  EMMii 

Reports .  Attached  as  Figures  D-8  thru  D-17  are  examples  of 
standard  EMMii  reports.  These  indicate  the  type  of  data  available 
to  the  trainer. 

Guides .  Each  course  contains  a  student  guide  that  includes 
the  course  text,  syllabus,  course  map,  list  of  lessons,  list  of 
topics,  all  training  objectives,  a  comprehensive  list  of 
references,  instructions  for  assembling  the  computer,  and 
instructions  in  case  the  computer  doesn't  respond  as  expected. 
Additionally,  the  brigade  operations  order,  student  handouts,  job 
aids  for  the  smart  book,  and  maps  or  templates  required  to 
complete  a  course  are  included  in  the  guide.  The  student  guide 
refers  the  student  to  existing  doctrinal  publications  but  does 
not  create  new,  or  modified,  doctrinal  publications.  One  of  the 
key  goals  of  the  courses  of  instruction  is  to  enable  the  student 
to  become  proficient  in  the  knowledge  in  the  doctrinal 
publications  and  learn  to  quickly  refer  to  the  publications  in  a 
field  environment. 

There  is  a  BSTS  instructor  guide  for  use  by  the  mentor, 
instructor,  or  trainer.  The  instructor  guide  includes  all  key 
information  in  the  student  guides  and  the  references  for  each 
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topic  to  guide  the  student  in  remediation.  Additionally, 
instructions  on  how  to  operate  in  the  EMMii  environment 
(scheduling  a  student,  monitoring  a  student's  progress,  reports, 
etc.)  are  included  in  the  guide. 

Delivery  Mechanisms .  The  CD  courseware  will  be  delivered 
via  486  PCs  equipped  with  a  16-bit  sound  card  and  speakers. 
Specifications  for  hardware  and  software  are  at  Appendix  B. 

Security.  Security  at  the  student  level  was  deemed 
unnecessary  except  when  the  student  exits,  to  ensure  data  is 
captured  for  the  database.  While  it  isn't  easy  for  a  student  to 
modify  his  answers  before  they  go  to  the  database,  it  is  not 
impossible. 

As  mentioned  earlier,  the  EMMii  allows  a  wide  range  of 
variation  in  access  to  data  among  users.  While  designed  for  LAN 
and  WAN  configuration,  the  system  was  modified,  at  the  request  of 
ARPA,  for  stand-alone  computer  operation.  The  stand-alone  mode 
was  fielded  to  the  two  test  brigades  and  a  train-the-trainer  (T3) 
course  for  unit  system  administrators.  Security  procedures  were 
adapted  for  unit  use  and  trained  during  the  T3  workshops. 
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Figure  D-9.  Screen' for  adding/editing  course  information. 
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Figure  D-10.  Screen  for  adding/editing  lessons. 


D-18 


Figure  D-12.  Viewing  a  student's  schedule 


Figure  D-13.  Viewing  a  student's  status  in  a  lesson 
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Figure  D-14.  Viewing  the  history  of  a  lesson. 


Figure  D -15.  Viewing  student  exam  answers. 
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Figure  D-16.  A  summary  of  student  opinions, 


Figure  D-17.  Viewing  one  student's  opinions. 
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Appendix  E 


BETA  Testing  Summary 

This  appendix  provides  a  summary  of  the  BETA  testing 
conducted  on  each  course  of  instruction  in  the  BDE-BSTS.  The 
more  detailed  data  that  were  collected  and  analyzed  during  the 
conduct  of  testing  are  available  elsewhere.  In  addition  to  the 
data  presented  in  this  appendix,  data  collected  for  each  course 
included:  specific  target  population  and  demographics;  post 
course  and  lesson  comments  in  the  categories  of  text,  doctrine, 
computer  based  instruction  (CBI) ,  and  tests;  test  scores  by 
lesson;  subject  and  lesson  summary;  recommended  actions  to  revise 
the  course;  and  actions  taken  to  revise  the  course  before  final 
production. 


The  general  target  population  description  included  student 
officers  that  were  AC  or  RC  officers  awaiting  follow-on 
assignment  having  completed  the  Infantry  Officer  Advanced  Course 
(lOAC) .  The  officers  had  BS  degrees  having  served  as  company 
commanders,  executive  officers,  rifle  platoon  leaders,  in  both 
light  and  heavy  forces.  The  average  number  of  CTC  rotations  was 
1.5,  with  an  even  split  between  both  NTC  and  JRTC.  Officers  had 
little  (.5  years)  to  no  battle  staff  experience.  Average  age  was 
31.5  years,  average  time  on  active  duty  was  5.5  years  with 
average  reserve  duty  time  2  years .  General  comments  from  all 
®ffic:ers  were  negative  on  use  of  traditional  correspondence 
courses,  highly  positive  to  small  group  instruction,  and  negative 
to  the  overall  quality  of  Army  doctrinal  publications.  Overall 
computer  literacy  was  low. 

[NOTE:  All  txines  and  scores/  from  this  point  on/  are  averages 
from  the  population  sample] 


Instructional  results  for  each  course  are  contained  in  the 
following  tables: 
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Table  El.  Common  Core 


PRE/POST  TEST  COMPARISON 

N= 

4 

SUBJECT : 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1 .  Brigade 
Battlefield 

75.5% 

89.5% 

19.0% 

YES 

2 .  Tactical 
Fundamentals 

68.0% 

80.0% 

18.0% 

YES 

3.  Staff 
Functions 

76.0% 

93.5% 

23.0% 

YES 

Common  Core  course  length  summary. 


Time  to  Complete:  TEXT 

DOCTRINE 

CBI 

TOTAL  TIME 


13.9 

1.75 

12.24 

27.89  hours 


Table  E2.  Executive  Officer. 


PRE/POST  TEST  COMPARISON  N=3 

SUBJECT : 

PRETEST 

FINAL  EXAM 

% IMPROVEMENT 

STANDARD 

MET? 

1. Chief  of 
Staff 

68.00% 

90.00% 

33.00% 

YES 

2 . Second  in 
Command 

70.00% 

88.33% 

26.19% 

YES 

61.00% 

91.00% 

49.18% 

YES 

Executive  Officer  course  length  summary. 

Time  to  Complete:  TEXT 

DOCTRINE 
CBI 

TOTAL  TIME 


12.42 

3.30 

29.00  hours 
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Table  E3.  Sl/Chaplain 


PRE/POST  TEST  COMPARISON  N=4 

SUBJECT : 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1 . Organization 
&  Planning 

75.00% 

100.00% 

34.23% 

YES 

2 .Manning 

63.00% 

91.00% 

44.44% 

YES 

3 . Sustaining 

80.25% 

97.75% 

21.81% 

YES 

Sl/Chaplain  course  length  summary. 


Time  to  Complete:  TEXT:  5.40 

DOCTRINE:  1.46 

CBI :  3.89 

TOTAL  TIME:  10.75  hours 


Table  E4.  S-2. 


PRE/POST  TEST  COMPARISON  N=3 

SUBJECT : 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1.S2  Organiza¬ 
tion  &  Mission 

56.0% 

89.3% 

59.5% 

YES 

2.  IPB 

35.0% 

90.0% 

157.0% 

YES 

3 . Intelligence 
Annex 

53.3% 

94.0% 

76.4% 

YES 

4. Analysis  & 
Dissemination 

60.3% 

92.0% 

52.6% 

YES 

S2  course  length  summary. 


Time  to  Complete:  TEXT:  6.60 

DOCTRINE:  8.28 

CBI:  9.84 

TOTAL  TIME:  24.70  hours 
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Table  E5.  S-3 


PRE/POST  TEST  COMPARISON  N=3  I 

SUBJECT : 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1 . Synchroniza¬ 
tion 

67.7% 

89.0% 

31.4% 

YES 

2 .  Combat  Spt 

'77.3% 

91.7% 

18.6% 

YES 

3 .  Supervise 
Execution 

68.3% 

90.0% 

31.7% 

YES 

S-3  course  length  suiranary. 


Time  to  Complete:  TEXT:  8.87 

DOCTRINE:  21.42 

CBI:  8.01 

TOTAL  TIME:  38.30  hours 


Table  E6.  S-3  Air. 


PRE/POST  TEST  COMPARISON  N=2  I 

SUBJECT : 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1 .  S3  Air 

Duties  &  A2C2 

63.5% 

100.0% 

57.5% 

YES 

97.5% 

56.0% 

YES 

3 .  Airspace 
Management 

42.5% 

91.0% 

114.1% 

YES 

4 .  Air  Ground 
Operations 

65.5% 

91.0% 

38.9% 

YES 

S-3  Air  course  length  siunmary. 


Time  to  Complete:  TEXT 

DOCTRINE 

CBI 

TOTAL  TIME 


13.01 

8.31 

29.3'S  hmiT-.c! 
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Table  E7 .  S-4 


PRE/POST  TEST  COMPARISON  N=3 

SUBJECT : 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1 .  Org  Tact 
Sustainment 

60.3% 

92.0% 

52.6% 

YES 

2. Tact  Sustain 
Ops 

55.0% 

89.7% 

63.0% 

YES 

3 .  Tact 

Sustainment 

Planning 

69.7% 

88.3% 

26.7% 

YES 

S-4  course  length  suimuary. 

Time  to  Complete:  TEXT: 

DOCTRINE : 
CBI: 
TOTAL  TIME: 


9.21 

10.30 

2.90 

22.41  hours 


Table  E8.  S-5. 


PRE/POST  TEST  COMPARISON  N=3 

SUBJECT : 

PRETEST 

FINAL  EXAM 

% IMPROVEMENT 

STANDARD 

MET? 

i  .Civil 
Affair  Ops 

60.00% 

86.00% 

44.13% 

YES 

2  .The 

Brigade  S5 

67.33% 

86.67% 

28.71% 

YES 

S-5  course  length  suinmary. 

Time  to  Complete:  TEXT: 

DOCTRINE : 
CBI: 
TOTAL  TIME: 


6.55 

4.42 

3.13 

14.10  hours 
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Tedile  E9.  Fire  Support  Officer 


PRE/POST  TEST  COMPARISON  N=4 

SUBJECT: 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1 . Organization 
&  Functions 

67.4% 

91.8% 

36.0% 

YES 

2. Fire  Support 
Planning 

88.5% 

22.0% 

YES 

3. Fire  Support 
Assets 

64.0% 

95.0% 

45.0% 

YES 

4 . Combat 
Operations 

54.0% 

93.3% 

73.0% 

YES 

Fire  Support  Officer  course  length  suinma2:y. 


Time  to  Complete:  TEXT 

DOCTRINE 

CBI 

TOTAL  TIME 


22.80 

20.80 

5.94 

49.50  hours 


Table  ElO .  Engineer . 


PRE/POST  TEST  COMPARISON  N=4 

SUBJECT : 

PRETEST 

FINAL  EXAM 

% IMPROVEMENT 

STANDARD 

MET? 

1 . Org  and 
Function 

64.50% 

95.25% 

47.70% 

YES 

2 .  Engr 
Planning 

78.50% 

96.25% 

22.60% 

YES 

3 .  Engr 

Assets 

68.50% 

96.25% 

40.50% 

YES 

4. Engr  Opns 

74.00% 

97.00% 

31.00% 

YES 

Engineer  course  length  summary. 


Time  to  Complete:  TEXT 

DOCTRINE 

CBI 

TOTAL  TIME 


11.40 

9.97 

2.10 

23.47  hours 
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Ted:>le  Ell.  Air  Defense  Coordinator 


PRE/POST  TEST  COMPARISON  N=3 

SUBJECT : 

PRETEST 

FINAL  EXAM 

% IMPROVEMENT 

STANDARD 

MET? 

l.Air  Def 
Planning 

64.00% 

92.00% 

45.03% 

YES 

2. Air  Def 

Cmd  &  Ctrl 

73.33% 

96.67% 

31.82% 

YES 

3 . ID  Friend 
or  Foe 

54.67% 

98.33% 

79.88% 

YES 

Air  Defense  Coordinator  course  length  summary. 


Time  to  Complete:  TEXT 

DOCTRINE 

CBI 

TOTAL  TIME 


5.50 
7.89 

8.50 

21.89  hours 


Table  E12.  Signal  Officer. 


PRE/POST  TEST  COMPARISON  N=3 

SUBJECT : 

PRETEST 

FINAL  EXAM 

% IMPROVEMENT 

STANDARD 

MET? 

1 . Signal 
Support 

60.00% 

86.00% 

44.13% 

YES 

2 . Signal 
Equipment 

67.33% 

86.67% 

28.71% 

YES 

Signal  Officer  course  length  summary. 


Time  to  Complete:  TEXT 

DOCTRINE 

CBI 

TOTAL  TIME 


2.47 

1.04 

4.41 

7.92  hours 
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Ted>le  E13 .  Chemical  Officer 


PRE/POST  TEST  COMPARISON  N=4 

SUBJECT : 

PRETEST 

FINAL 

EXAM 

%  IMPROVEMENT 

STANDARD  MET? 

1 .  Org  & 

Duties 

61.25% 

91.75% 

50% 

YES 

2.  NBC 
Operations 

62.0% 

92.0% 

48% 

YES 

3 .  US  &  Threat 
Equipment 

44.5% 

87.25% 

96% 

YES 

Chemical  course  length  summary. 

Time  to  Complete:  TEXT: 

DOCTRINE : 
CBI: 
TOTAL  TIME: 


15.40 

9.18 

4.22 

28.86  hours 


